In a ring topology interconnection network, all the devices in the subsystem are arranged in a ring or loop configuration, and typically all communication is unidirectional around the ring. As a result, each device in the ring has exactly one other device immediately “before” and exactly one device that is immediately “after” it.
In many ring-topology networks, the ring is physically, and electrically, broken by each device in it. In these networks, the ring itself is made up of a number of ring segments that connect the devices in the ring. The number of segments equals the number of devices in the ring. Communication on each segment is unidirectional and each has a driver end where exactly one device drives information onto the segment, and a one receiver end where exactly one device receives information from the driving device. From the perspective of a device on the ring, the segment of the ring on which it receives information is called its input segment, while the segment onto which it drives information is called its output segment.
It is typical for one device in the ring to be special in that it is the master device. It is often and synonymously referred to as the controller. For example in memory subsystems constructed from such a ring-topology interconnect, the master or controller is typically responsible for both issuing read and write and other commands to the memories on the ring under its control, and for orchestrating data communication around the ring. The other devices in the ring are slaves and are typically but not exclusively memories. These slaves respond to commands from the controller sent over the ring and, in the case of read and other commands, send data back to the controller, also over the ring. Generally, the timing of communication on the ring is determined by the controller and not freely initiated by the slaves.
Ring topologies are popular because they are conceptually simple, generally extensible and can require only a few pins per device. Also, since only point-to-point communication is used, issues of signal integrity and segment length are much more tractable than with many other interconnect topologies. However, three significant drawbacks evident with this class of ring-topology interconnect relate to latency, failure tolerance and power consumption.
With regard to latency, for all networks in this class, the only path from a controller to an accessed device and back to the controller is through all the other devices in the ring. As such, in the case of a controller accessing a slave and commanding it to respond by delivering data back to the controller, the latency of the operation has two primary components: the intrinsic latency of the operation within the slave, and the communication latency around the ring. Since the ring goes through each of the devices, each of these devices will present some small latency to the commands and data flowing round the ring. In addition the ring segments themselves can add a non-trivial amount to communication latency.
An inherent characteristic of this class of ring-topology interconnects then is that total latency around the ring is the sum of all the latency through the devices plus the latency of the ring segments connecting those devices. The total latency around the ring is important since the total slave read access time from a controller to a device and back is the sum of the total ring latency and the slave device's internal read access latency. Reducing total ring latency is an important perquisite for circumstances in which total access latency is a significant component contributing to system-level performance. Thus in some known examples, although a ring topology network may be able to support a large number of devices, on the basis of latency alone, it may be impractical for all applications that involve more than a small handful of devices.
Another significant problem with ring-topology networks is that they are fault-intolerant. If any of the devices or segments on the ring fails, communication around the ring is impossible, even if the other devices are still completely functional. This characteristic of ring-topology networks is significant in some system environments and not others. However, for memory systems in particular, there are many system applications that impose the requirement that the memory subsystem be able to operate flawlessly even in the presence of a single bad device.
A third significant problem with ring-topology networks is that every device in the ring expends power to communicate each packet around the network. Depending on the signaling technology, every slave in the ring expends power on every command packet (regardless of the receiver) and on every read data packet (regardless of the source). In contrast, with some other network topologies, slave devices only expend power to communicate read data back to the controller. So, even though the power expended by a slave per bit transmitted can be less given the point-to-point communication, the total power efficiency of a ring-topology network can be significantly worse than some non-ring alternatives.
There have been previous attempts to mitigate or avoid some of the negative characteristics of ring-topology networks. These efforts fall into four categories. In the first category, the total latency is minimized by keeping the per-device latency as small as possible. This is the approach disclosed in U.S. Pat. No, 5,778,419 and in the RamLink memory interface (IEEE Standard for High-bandwidth Memory Interface Based on Scalable Coherent Interface(SCI) Signaling (RamLink), EEE Standard 1596.4-1996). Another approach is to limit the total latency around the ring by artificially limiting the number of devices in the ring. This evident in the U.S. Pat. No. 5,778,419, where the number of devices in the ring is limited to 5 (one master and 4 slaves).
A third technique is to create a hierarchy of rings, in which each of the devices on the primary ring (i.e. the ring controlled the overall master) is an agent acting as a master for a sub-ring of slaves (See FIG. 1). Commands targeted at specific slaves are intercepted by the appropriate agent and translated into commands on the single specific sub-ring that includes the targeted slave. A fourth method of dealing with the deficiencies is to have a ring of agents that in turn control non-ring based devices. For example in FIG. 2, each agent controls a traditional parallel memory subsystem.
These previous solutions either fail to fully take advantage of the inherent advantages of a ring topology or fail to adequately address the problems of ring topology networks described above. For example, the ring of rings topology has been calculated to reduce the communication latency for a system with n slaves to 2×sqrt(n) plus any additional scheduling delay in the agent. Reducing the per-slave delay reduces the constant of proportionality, but the total communication latency still remains proportional to the number of devices in the ring. Limiting the number of devices in the ring to a small number caps the total communications latency, but also limits the utility of the interconnect in larger systems.
Some approaches to improve ring network performance involve the use of multi-chip modules. Multi-chip modules are packaging structures in which more than one integrated circuit is housed together with a common set of signal pins to the board or carrier on which the multi-chip module is mounted. A more precise definition is provided in the Detailed Description section. The straightforward application of integrated circuits in multi-chip modules to a ring-topology interconnect is to serially connect all the devices in the multi-chip module into the ring. The topology and latency of this configuration are as if each integrated circuit was in its own package and the packages were serially connected into the ring. The result would be a package cross section comparable to that shown in FIG. 3, where one can see the link coming into the lowest device on bonding wires, leaving the lowest device and entering the one above it, and so up the stack, culminating with bonding wires connecting the topmost device with the package to form the outbound ring segment.
The configuration of FIG. 3 is less than ideal for several reasons. First, although these devices appear as a single package, they present logically as five devices, presenting five devices worth of latency. Second, this particular arrangement of incoming and outgoing segments on opposing sides of devices may not be convenient. In general, a multi-chip module of n devices presents physically as one device but logically as n devices, This configuration fails to adequately take advantage of the close proximity of the integrated circuits in the multi-chip package to address some of the limitations of large rings where a multi-chip module approach is not taken.